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Executive Summary 


1. The purpose of evaluation ts to improve health programmes 
and the health infrastructure for delivering them and to guide 
the allocation of resources in current and future programmes. 
Evaluation is a systematic way of learning from experience and 
using the lessons learned to improve current activities and pro- 
mote better planning by careful selection of alternatives for future 
action. It should thus be a continuing process aimed at rendering 
health activities more relevant, more efficient, and more effective. 
Its application calls for an open mind capable of constructive 
criticism in order to lead to sound judgement. 


2. Health programme evaluation ts part of a broader mana- 
gerial process for national health development. This process 
consists of an interlinked sequence of events including policy 
formulation, programme formulation and budgeting, programme 
delivery through various services and institutions, evaluation, 
and reprogramming as necessary, with information support through- 
out. This process is well illustrated in WHO?’s publication on the 
managerial process for national health development in support of 
strategies for health for all by the year 2000.1 The place of evalu- 
ation in the managerial process for national health development 
is illustrated in Fig. 1 (see page 13 below). 


3. The responsibility for evaluation devolves on the individuals 
and groups who are responsible for the development and applica- 
tion of the managerial process for national health development 
in the country concerned. 


1 Managerial process for national health development. Guiding principles, 
Geneva, World Health Organization, 1981 (“Health for All” Series, No. 5). 
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4. Indicators are used in the evaluation process. These are 
variables which help to measure changes. Criteria are also used. 
These are standards by which actions are measured. Where no 
suitable indicators or criteria are available, pertinent questions 
should be asked concerning the activity to be evaluated. 


5. Evaluation has to be supported by valid, relevant and sensi- 
tive information. 


6. The evaluation process is intended for use in a flexible 
manner and must be adapted to the circumstances in which it 
is to be used. It consists of the following components: 

— Specify particular subject for evaluation 

— Ensure information support 

— Verify relevance 

— Assess adequacy 

— Review progress 

— Assess efficiency 

— Assess effectiveness 

— Assess impact 

— Draw conclusions and formulate proposals for future action. 

7. The table on page 27 illustrates the application of the 
evaluation process in the course of applying the various compo- 
nents of the managerial process for national health development. 
(The first two components listed in the previous paragraph— 


specify particular subject for evaluation and ensure information 
support—form a part of all others.) 


8. The frequency with which the various components men- 
tioned in paragraph 6 above can be applied will vary, ranging 
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from relatively short intervals for the assessment of progress and 
efficiency to much longer intervals for the assessment of effective- 
ness and impact. 


9. The more detailed guiding principles that follow are a first 
step towards better integration of evaluation into the managerial 
process for national health development. 


10. It should be understood that these guiding principles for 
evaluation are, of necessity, general in nature, since they are 
meant to meet the requirements of all Member States, in all fields 
of public health. Yet they endeavour to state principles and to 
outline practical steps in a manner applicable to diversified na- 
tional and public health fields. 
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I. Introduction 


1. In 1978, at the request of Member States, WHO prepared 
provisional guidelines for health programme evaluation for use 
both by countries for their programmes and activities, and by 
WHO for its collaborative programmes. These general guidelines, 
which were endorsed by the Thirty-first World Health Assembly, 
have been adapted in the text that follows with a view to integrat- 
ing evaluation into the overall managerial process for national 
health development. 


2. Evaluation, in the context of that managerial process, should 
be a continuing process aimed mainly at correcting and improving 
actions in order to render health activities more relevant, more 
efficient and more effective. In this perspective evaluation calls 
for an open mind capable of constructive criticism. It further 
requires willingness to communicate freely with professional 
peers and other persons or groups concerned at various policy 
and operational levels of health systems as well as other related 
social and economic systems, whether national or international. 


3. Evaluation implies judgement based on careful assessment 
and critical appraisal of given situations, which should lead to 
drawing sensible conclusions and making useful proposals for 
future action. It should not be seen as “pronouncing a sentence” 
in a judicial sense. The judgement has to be based on valid, rele- 
vant and sensitive information that is readily and easily available 
and that is provided to all those who need it. 


4. The process outlined in these guiding principles should 
therefore be regarded as a model to provide a systematic basis 
for evaluation. The use of such a model should help to free the 
mind to concentrate on the judgement required. 
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5. It should be noted that the guiding principles are—as the 
words imply—intended for flexible use, and should not be con- 
sidered as a formal manual. The process of evaluation requires 
adaptation of these guiding principles to each specific situatio 
and would suffer in its usefulness if too rigid a system were a: 
posed. It should also be noted that the guiding principles yw 
follow are a first step towards better integration of the evaluatio 
process into the overall managerial process. ; 


II. General Principles 


6. Evaluation is a systematic way of learning from experience 
and using the lessons learned to improve current activities and 
promote better planning by careful selection of alternatives for 
future action. This involves a critical analysis of different aspects 
of the development and implementation of a programme and the 
activities that constitute the programme, its relevance, its formula- 
tion, its efficiency and effectiveness, its costs and its acceptance 
by all parties involved. 


7. Thus, the purpose of evaluation in health development is 
to improve health programmes and the services for delivering 
them, and to guide the allocation of human and financial resources 
in current and future programmes and services. It should be used 
constructively and not for the justification of past actions or 
merely to identify their inadequacies. It is essential to perceive 
evaluation as a decision-oriented tool, and to link the evaluation 
process closely with decision-making, whether at the operational 
or the policy level. The very process of carrying out an evaluation 
can be just as important as the conclusions drawn, since involve- 
ment in the process itself often induces a better understanding of 
the activities being evaluated, and a more constructive approach 
to their implementation and to any future action required. 


8. Evaluation, difficult in any field, presents particular prob- 
lems in health work owing to the very nature of the activities, 
which often do not lend themselves’ easily to the measurement 
of what has been attained against predetermined, quantified 
objectives. It is therefore often unavoidable to apply qualitative 
judgement, supported, wherever possible, by reliable, quantified 
information. Account has to be taken of the intricate interrelation- 


Purpose of 
evaluation 


Constraints 
of evaluation 
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Place of 
evaluation in the 
health devel- 
opment process 


12 


ships between the health and other social and economic sectors. 
Changes in a health situation are often brought about by elements 
outside the health sector, making evaluation, particularly of effec- 
tiveness and impact, even more difficult. This accentuates the 
need to define reliable and sensitive indicators for identifying 
changes in health status or in the improvement of health care 
delivery. The problems related to the collection of information 


for these indicators have repercussions on the feasibility of their 


use. / 


9. Another constraint that is often encountered is a certain 
in-built resistance in principle to accepting evaluation and its 
results as a valid management tool. Self-defence often leads to 
rejection; it is not difficult to prove that an evaluation process 
was not “scientific enough to provide a “sound” basis for 
making programme decisions, or to discredit the evaluation results 
by challenging the validity of certain criteria used. 


10. The purpose of the managerial process for national health 
development is to build up the health system in a rational and 
systematic way. Health programme evaluation is part of the man- 
agerial process for national health development, as described in 
the volume already mentioned.” The diagram opposite (see Fig. 1) 
illustrates the place of evaluation in the overall managerial pro- 
cess. 


11. In order to carry out evaluation as part of the managerial 
process, it has to be planned for when working out that process 
in each country; for example, evaluation of broad programming 


‘For more details on indicators and the related question of information 
collection, see: Development of indicators for monitoring progress towards 
health for all by the year 2000, Geneva, World Health Organization, 1981 
(“Health for All’’ Series, No. 4). 


? Managerial process for national health development. Guiding principles, 
Geneva, World Health Organization, 1981 (“Health for All” Series, No. 5). 
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has to be taken into account before detailed programming is 
initiated. 


12. There is a great variety of national health systems. Thus, Framework for 
national health policies vary according to political, economic and evaluation 
social systems. Most countries have plans to develop their health 
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systems, some more formal than others. Some countries have 
health programmes with well-defined objectives and targets. All 
have various types of health services and institutions as well as 
different types of professional and other health workers. 


13. Governments are taking an increased interest in health 
development. In some countries the government provides all 
health care. In others, various types of social security and health 
insurance schemes exist, sometimes alongside government health 
care and sometimes with their own health institutions. In addition, 
in some countries health care is provided by voluntary agencies 
and by the private sector. 


14. The process suggested in these guidelines for evaluation 
as part of the managerial-process is general in nature and has to 
be adapted for use within the large variety of national health 
systems. A model of one such system has been used for con- 
venience. 


15. Such a model of a government-directed health system of 
a medium-sized country would include local, district and central 
levels. At the local level some or all of the essential elements of 
primary health care would be provided.* At district level there 
would be a district health office to manage all the health affairs 
of the district, a district hospital with an outpatient dispensary, 
a public health laboratory, and schools for nurses and auxiliaries. 
Sometimes the environmental health services would be managed 
from the district health office; sometimes they would be managed 
by other sectors. At central level there would be a ministry of 
health or equivalent body dealing with health planning, the 
management of the government health services and institutions 
and certain environmental services, large general and specialized 
hospitals, medical schools, as well as schools for nurses and other 
professional health workers, and central public health laboratories. 


' Alma-Ata 1978. Primary health care, Geneva, World Health Organization, 
1978 (“Health for All” Series, No. 1), page 4, para. 3. 
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16. Health systems are undergoing profound change in many 
countries. Numerous countries have introduced the managerial 
process for national health development or an equivalent process, 
and this will no doubt facilitate evaluation once programmes have 
been well defined. Other countries will have to define in other 
ways programmes and their component parts that are to be evalu- 
ated. The establishment and management of health services and 
institutions for programme implementation are often of particular 
importance, calling for specific kinds of evaluation. 


17. As for the responsibility for evaluation, the principle 
whereby evaluation should be conducted as an integral part of 
the various steps of the overall managerial process implies that 
individuals and groups responsible for the development and appli- 
cation of that process at the various policy and operational levels 
also carry responsibility for its evaluation. 


18. Thus, to revert to the model outlined above, responsibility 
will lie at the local level with those in charge of primary health 
care. Communities themselves will also carry responsibility for 
evaluating the appropriateness of the services they receive and 
the satisfaction of community members with the health care 
available to them. At district level, responsibility will be with 
directors of district hospitals, public health laboratories, environ- 
mental health services, training schools and, finally, the district 
health officer; at central level, it will be with the directors of the 
various hospitals, deans of medical and nursing schools, central 
public health laboratories, programme directors in the ministry 
of health, the director-general of health services in the ministry 
of health, the minister of health, and, possibly, the state control- 
ler, the minister of finance, the parliament or other governmental 
institutions, as appropriate in each country. 


19. An important part of the responsibility at each level is to 
ensure that other individuals and groups concerned at the same 
level or at other levels, whether more centrally or more periph- 
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erally located, are in a position to contribute to the evaluation, 
are kept informed of its results, and are requested to take appro- 
priate action. This is essential in order to initiate or maintain the 
dialogues required among these persons. 


20. As some of the evaluation issues might be of a sensitive 
or controversial nature, the evaluation activity as well as its find- 
ings should be shared at regular intervals by all involved in the 
process. To facilitate this task a current record should be main- 
tained of the important decisions and options, of the reasons for 
selecting one of several courses of action, and of the actual events 
and intermediate results as they occur. 


21. The following are the main features of the components 
of the evaluation process: 


(1) Relevance relates to the rationale for adopting health 
policies in terms of their response to social and economic pol- 
icy; and to having programmes, activities, services or institu- 
tions, in terms of their response to essential human needs and 
social and health policies and priorities. 


(2) Adequacy implies that sufficient attention has been paid 
to certain previously determined courses of action, such as the 
various issues to be considered during broad programming. 


(3) Progress is concerned with the comparison of actual with 
scheduled activities, the identification of reasons for achieve- 
ments or shortcomings, and indications for remedies for any 
shortcomings. The purpose of a progress review is to facilitate 
the monitoring and operational control of ongoing activities. 
In this context monitoring is the day-to-day follow-up of an 
activity during its implementation to ensure that operations are 
proceeding as planned and are on schedule. It keeps track of 
ongoing activities, milestones achieved, personnel matters, sup- 
plies and equipment, and money spent in relation to budgets 
allocated. 
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(4) Efficiency is an expression of the relationships between 
the results obtained from a health programme or activity and 
the efforts expended in terms of human, financial and other 
resources, health processes and technology, and time. The 
assessment of efficiency is aimed at improving implementation, 
and adds to the review of progress by taking account of the 
results of monitoring. Under this heading, a check is also made 
on such matters as the appropriateness of existing plans of 
operations, work schedules, methods applied, manpower used, 
and the adequacy and use of financial resources, with a view 
to improving them, if necessary, at the least cost. 


(5) Effectiveness is an expression of the desired effect of a 
programme, service, institution or support activity in reducing 
a health problem or improving an unsatisfactory health situa- 
tion. Thus, effectiveness measures the degree of attainment of 
the predetermined objectives and targets of the programme, 
service or institution. The assessment of effectiveness is aimed 
at improving programme formulation or the functions and 
structure of health services and institutions through analysis 
of the extent of attainment of their objectives. Where feasible, 
the extent of attainment should be quantified. Where this is 
not feasible, a qualitative analysis of the relevance and useful- 
ness of the achievement has to be performed, however sub- 
jective and impressionistic such an analysis may be, until a 
more precise way of measuring is developed. The evaluation 
of effectiveness should also include an assessment of the satis- 
faction or dissatisfaction expressed by the community con- 
cerned regarding the effects of the programme, service or insti- 
tution. If possible, cost-effectiveness should be assessed and a 
cost-benefit analysis made. 


(6) Impact is an expression of the overall effect of a pro- 
gramme, service or institution on health and related socio- 
economic development. The assessment of impact is thus aimed 
at identifying any necessary change in the direction of health 
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programmes so as to increase their contribution to overall health 
and socioeconomic development. 


22. The description of these components of evaluation would 
be incomplete without reference to the frequency with which they 
can be considered. While evaluation is a continuing process, its 
results have to be summarized and reported on at given times or 
specified intervals. It will no doubt be found easier to summarize 
the assessment of progress and efficiency, say, once a year than 
to assess effectiveness, for which a longer time-span might be 
required. This results from the need to identify significant changes 
in the specific health situation that are indicative of a programme’s 
effectiveness. An even longer time-span is likely to be required 
in relation to the assessment of impact; at least five years from 
the inception of a programme may be required. 


23. Indicators and criteria are used as aids throughout the 
evaluation process. They can also be used as aids for planning 
and programming. 


24. Indicators are variables that help to measure changes. 
They are evaluation tools which can measure change directly or 
indirectly. For example, if the objective of a programme is to 
train a certain number of auxiliary health personnel annually, a 
direct indicator for evaluation could be the number of such per- 
sonnel actually trained each year. If the subject of evaluation is 
the result of a programme aimed at improving the level of health 
of a child population, it may be necessary to assess any improve- 
ment by using several indicators that could indirectly measure a 
change in this level. Such indicators could be the nutritional status 
as illustrated by weight in relation to height, the rate of immuni- 
zation, learning capacity, age-specific mortality rates, disease- 
specific morbidity rates, and disability rates of a child population. 


25. When selecting indicators, full account has to be taken of 
the extent to which they are valid, reliable, sensitive, and specific. 
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26. Validity implies that the indicator actually measures what 
it is supposed to measure. Reliability implies that even if the 
indicator is used by different people at different times and under 
different circumstances, the results will be the same. Sensitivity 
means that the indicator should be sensitive to changes in the 
situation or phenomenon concerned. However, indicators could 
be sensitive to more than one situation or phenomenon. Specificity 
means that the indicator reflects changes only in the situation or 
phenomenon concerned. For example, the infant mortality rate 
is a sensitive, although very crude, indicator of the level of health 
of a child population. It is a direct measure of death and only an 
indirect measure of health, but within this limitation it is both a 
valid and a reliable indicator. It is not specific, however, in 
relation to any particular health action, since its reduction can 
result from a large number of factors related to social and econo- 
mic development, including health development, and can rarely 
be attributed to any one health action. 


27. Health and health-related indicators, often considered in 
various combinations, are used in particular to assess effective- 
ness and impact. A vast number of indicators exists. They can be 
. placed in groups relating to health policy, socioeconomic condi- 
tions, provision of health care, and health status. While indicators 
of health policy and of the provision of health care are useful 
mainly for assessing effectiveness, social and economic indicators 
and indicators of health status have to be used in the final analysis 
to assess impact. The following are examples of such indicators: 


(1) HEALTH POLICY INDICATORS’ 


— High-level political commitment to health for all 


— Allocation of adequate resources for primary health care 


‘For further explanations, see: Development of indicators for monitoring 
progress towards health for all by the year 2000, Geneva, World Health 
Organization, 1981 (“Health for All” Series, No. 4). 
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— Degree of equity of distribution of resources 
_— Level of community involvement in attaining health for all 


— Establishment of a suitable organizational and managerial 
framework for the national strategy for health for all 


— Practical manifestations of international political commit- 
ment to health for all 


(2) SOCIAL AND ECONOMIC INDICATORS 


— Rate of population increase 


— Gross national product (GNP) or gross domestic product 
(GDP) 


— Income distribution 
— Work availability 
— Adult literacy rate 


— Adequacy of housing expressed as number of persons per 
room 


— Per capita energy availability 


(3) INDICATORS OF THE PROVISION OF HEALTH CARE 


— Availability 

— Physical accessibility 

— Economic and cultural accessibility 
— Utilization of services 

— Indicators for assessing quality of care 
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(4) INDICATORS OF COVERAGE BY PRIMARY HEALTH 
CARE 
— Level of “health literacy” 


— Availability of safe water in the home or within short 
walking distance 


— Adequate sanitary facilities in the home or immediate 
vicinity 
— Access of mothers and children to local health care 


— Birth attendance by trained personnel 


— Percentage of children at risk immunized against the 
major infectious diseases of childhood 


— Availability of essential drugs throughout the year 
— Accessibility of referral institutions 


— Ratio of population to different kinds of health worker in 
primary health care and at referral levels 


(5) HEALTH STATUS INDICATORS 


— Percentage of newborn infants with birth weight of a least 
2500 g 


— Percentage of children that have a Ent for age that cor- 
responds to specified norms 


— Indicators of the psychosocial development of children | 
LEA 00 


— Infant mortality rate v8) 


— Child mortality rate 
— Under-5-year mortality rate 


— Life expectancy at a given age 
21 
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— Maternal mortality rate 

— Disease-specific mortality rates 
— Disease-specific morbidity rates 
— Disability rates 


— Indicators of social and mental pathology, such as rates of 
suicide, drug addiction, crime, juvenile delinquency, alco- 
holism, excessive smoking, obesity and consumption of 
tranquillizers 


28. Criteria are standards by which actions are measured. 
Criteria may be technical or social, technical criteria being nor- 
mally highly specific to programmes. For example, a technical 
criterion for the guarantee of the safety of drinking-water would 
have to be a certain technical standard for the purity of water. A 
social criterion for the guarantee of the continuation of the water 
supply could be the existence of a community organization for 
the maintenance of the supply. 


29. The main purpose of criteria is to provoke thought lead- 
ing to judgement. Such judgement can rarely be arrived at only 
through adding up numerical values resulting from certain quan- 
tified evaluation criteria. 


30. Nevertheless, criteria for the evaluation of health pro- 
grammes should be quantified wherever possible, although in 
practice this will not always be possible, particularly where social 
criteria are concerned. It is therefore often necessary to resort to 
an evaluation based on qualitative rather than quantitative assess- 
ment. 


31. Another very important attribute of both indicators and 
criteria is the availability of the information required: that is, it 
should be possible to obtain the data required without undue 
difficulty. Indicators and criteria should be included in the pro- 


Health programme evaluation 


gramme at the planning stage, so that the information require- 
ments can be determined early on. 


32. It has to be realized that there will be health activities for 
the evaluation of which no suitable indicators and criteria may be 
available. In these cases, pertinent questions should be asked 
concerning the activity to be evaluated. Answers to these ques- 
tions will help to guide evaluation and will, in turn, help to define 
and refine indicators and criteria. For example, the following 
illustrative questions could be asked: 


(1) Has “health for all” received endorsement as policy at 
the highest official level? 


(2) Have mechanisms been formed or strengthened for 
involving people in the national strategy for health for all? 


(3) How much of the GNP is spent on health, and how much 
of this is spent on various population groups or geographical 
areas, such as urban and rural areas? 


(4) How much of the health budget is spent on local health 
care? 


(5) What percentage of health expenditure in developed 
countries is transferred to support strategies for health for all 
in developing countries? 


(6) Is safe water available in the home, or how far away is it 
available in terms of walking distance? 


(7) Are there adequate sanitary facilities in the home or im- 
mediate vicinity? 

(8) What proportion of children is immunized against the 
major infectious diseases of children? 


(9) What walking or travel time is required to obtain local 
health care? 
(10) Is childbirth attended by trained personnel? 
23 
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(11) Is the nutritional status of children adequate? 


(12) Do large numbers of children die during their first year 
of life? 


(13) Is there a high proportion of elderly people? 


(14) Can most adults read and write, and is there a great 
difference between women and men in this respect? 


(15) What is the GNP per head? 


33. Evaluation has to be based on valid, relevant and sensitive 
information. There is often an excess of only marginally relevant 
and sensitive information. Often the most needed information is 
not available and its collection can be extremely costly. For the 
above reasons an approach of high selectivity should be adopted 
and only that information should be collected which has been 
identified as being really crucial for the issue being evaluated. 


34. The types of information required may include political, 
social, cultural, economic, environmental and administrative fac- 
tors influencing the health situation as well as mortality and mor- 
bidity statistics. Any of this information may form the basis of 
indicators and criteria for use in the various steps of the evalua- 
tion process. The information required may also concern health 
and related socioeconomic policies, plans and programmes, as 
well as the extent, scope and use of health systems, services and 
institutions. Thus, the information sought may be historical, social, 
political, economic, scientific, technological, demographic, epi- 
demiological, organizational, legislative, or related to inventories 
of resources. 


35. Information collection is least costly when the informa- 
tion emanates from the operation of the programmes or services 
of a health system. The required flow of information will vary by 
countries, e.g., from the periphery through various administrative 
levels to the ministry of health, with subsequent feedback from 
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the centre to local health units. It may be necessary to have spe- 
cific information collected in the absence of well developed health 
information support to the managerial process.’ This in itself 
could help to strengthen such information support, including a 
well organized reporting system. 


1Methods of information collection are described in: Development of 
indicators for monitoring progress towards health for all by the year 2000, 
Geneva, World Health Organization, 1981 (“Health for All’ Series, No. 4). 
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III. The process of evaluating 
health policies, programmes, 
services and institutions 


The evaluation 36. The process of evaluation consists of the following com- 
process ponents: 
— specify the particular subject for evaluation 
— ensure information support 
— verify relevance 
— assess adequacy 
— review progress 
— assess efficiency 
— assess effectiveness 
— assess impact 
— draw conclusions and formulate proposals for future action. 
37. The above process is intended for use in a flexible manner, 
and must be adapted to the circumstances in which it is to be 
used. The table opposite illustrates the application of the process 


in the course of applying the various components of the mana- 
gerial process for national health development. 


38. Those responsible for evaluation should set the evalu- 
ation process in motion giving due consideration to the above 
components. The process should start at local level, moving pro- 
gressively towards the central level. 
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COMPONENTS OF MANAGERIAL COMPONENTS OF EVALUATION PROCESS 
PROCESS FOR NATIONAL 
HEALTH DEVELOPMENT 


Policy formulation : Verify relevance 
Assess adequacy of problem definition 


Broad programming : Verify relevance of programmes 
Programme budgeting : Assess adequacy of formulation 
Master plan of action : Assess adequacy of formulation 
Detailed programming : Review progress 
Assess adequacy of formulation 
Implementation : Review: progress 
efficiency 
effectiveness 
impact 
Note: 
It should be noted that these : In consequence, since the evaluation pro- 
components are usually being cess is integral to the managerial process, 
applied at the same time with its various components too are usually 
respect to different pro- carried out at the same time with respect 
grammes, possibly in diffe- to different programmes in different parts 
rent parts of the country and of the country and at different policy and 
at different policy and opera- operational levels. 


tional levels. 


39. The specification of the particular subject for evaluation Specify the 
is carried out by answering the following questions: subject for 

(1) What is to be evaluated? That is: should the subject to evaluation 

be evaluated be a programme, e.g., maternal and child health; 

a service, e.g., water supply and sewage disposal, or the use of 

ambulances and other transport; a local health service deliver- 

ing a number of programmes; or an institution, such as a health 

centre, a hospital or a training institution? Make sure that the 

subject is really worth evaluating in terms of its size or poten- 

tial importance. It would not be justifiable to evaluate minor 

activities outside the broader programme, service or institution 

of which they form a part. . 
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(2) At what organizational level or levels is the evaluation to 
be made? 


(3) What is the purpose of the evaluation? Is it in support of 
the annual budget allocation, or of a new development plan, 
or for some ad hoc purpose? 


(4) What are the constraints that could limit the possibility of 
evaluation or restrict its scope? 


(5) What options exist for decisions to be taken in the light 
of the evaluation results? These could include, to give some 
examples: continue unchanged; modify; terminate; transfer to 
another administrative level; accelerate pace of implementa- 
tion; amalgamate with another programme or service; reassess 
budget; request extrabudgetary funds; or reconsider whether 
implementation requires international collaboration, e.g., as 
part of technical cooperation among developing countries 
(TCDC), or with WHO or any other international or bilateral 
organization. The answers to the above are of particular rele- 
vance to the replanning and programming of activities. 


(6) To whom should the evaluation results be reported? 
Possible answers could include the group or individual respon- 
sible for carrying out the related component of the managerial 
process or for managing the service or institution concerned. 
More formally it could include the local government, the district 
health officer and the ministry of health. In other words, evalu- 
ation results should be reported both within the group respon- 
sible for the activity concerned and to the next link in the 
chain of responsibility. In turn, those responsible for the next 
link should report back on their assessment of the evaluation 
results presented to them. Efforts should be made to present 
the results in the most economical way, avoiding costly forms 
of publication. 


Ensure informa- 40. The use of information permeates the whole of the evalu- 
tion support ation process. It is therefore necessary to ensure adequate infor- 
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mation support throughout the process, so that this will not sub- 
sequently be delayed for lack of such information. The best way 
to ensure information support is to specify the information re- 
quirements at the planning stage. Answers to the following three 
questions will be of assistance in ensuring such support in any 
evaluation effort: 


(1) What are the information requirements? 
(2) What are the available sources of information? 


(3) Is the available information adequate or is it necessary 
to obtain further or more precise information from additional 
sources, or by carrying out special studies and surveys? 


41. The information requirements should be identified in the 
light of the needs of the various components of the evaluation 
process. For example: 


(1) Subject of evaluation—summarize the answers to the 
questions in paragraph 39 above, taking account of the time- 
span to be covered by the evaluation. 


(2) Relevance—specify the health policy basis in relation to 
social and economic policy, and programme objectives in rela- 
tion to health policy. 


(3) Adequacy of problem definition—make sure that a state- 
ment exists that defines the problem for the solution of which 
the process or programme is being implemented. 


(4) Adequacy of programme or project formulation—make 
sure that there is a list of objectives and targets; a description 
of the approaches being used to attain the objectives and meet 
the targets; a summary of the resources involved, including 
personnel and budgeting and financial information; and a sched- 
ule of activities, including milestones or checkpoints and their 
due dates. Infer information requirements from the criteria to 
be used or questions to be asked. 
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(5) Progress—verify whether information is available on the 
use of resources. Infer information requirements from the 
indicators to be used or questions to be asked. 


(6) Efficiency—identify the results of the programme or pro- 
ject or make sure that they can be summarized. Specify the 
indicators and criteria to be used or questions to be asked and 
infer information requirements from these. 


(7) Effectiveness—verify the availability of information on 
the health problem or situation concerned before the start of 
the programme, at earlier stages of its implementation, and at 
the time of the evaluation. Infer information requirements from 
the indicators and criteria to be used or questions to be asked. 


(8) Impact—verify the availability of information on the 
overall health and related socioeconomic situation concerned 
before the start of the programme, at earlier stages of its im- 
plementation, and at the time of evaluation. Select the perti- 
nent information, and infer further information requirements 
from the indicators and criteria to be used or questions to be 
asked. 


(9) Conclusions and proposals for future action—refer to 
any conclusions and proposals for action from preceding evalu- 
ation efforts. 


42. The following are examples of sources of information 


which will usually be required and will be available: 


— government reports on the political, economic and social 
situation; 


— policy documents (health plans, development plans, reports 
concerning the overall managerial process, recent important 
statements by political leaders, legal documents); 


— periodic reports of the ministry of health; 
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— epidemiological information (from official national health 
statistics services, reports, research findings, etc.); 


— demographic information, in particular vital statistics; 


— data on the resources used and activities carried out in the 
services and institutions; 


—health programme formulation documents, and plans of 
action for implementing programmes or establishing institu- 
tions and services. 


43. At this stage, it is wise to analyse the adequacy of avail- 
able information. Where justified, a preliminary screening should 
be made of the data at hand to decide whether they are (a) usable 
and (b) sufficient, both qualitatively and quantitatively, and 
whether extra data should be collected. If there is a need for 
extra data, it has to be decided whether they can be obtained 
easily from available sources, or whether a special study or survey 
has to be carried out. 


44. When information cannot easily be obtained from avail- 
able sources and it has therefore been decided to carry out a special 
survey, the collection of the necessary data must be properly 
planned and implemented. The cost of collecting the data has to 
be carefully considered, because this can sometimes be very high 
in relation to the benefit from the survey. Wherever possible, the 
live collection of data—that is, directly from the people con- 
cerned—is to be preferred to collection by correspondence. Such 
live communications have an importance which goes beyond in- 
formation-gathering and facilitates the very judgement inherent 
in the evaluation process. 

45. Inlarge-scale surveys, or with a view to reducing the costs 
of data collection, it may be necessary to limit the amount of data 
gathered, e.g., by taking samples. In this case, the sampling pro- 
cedure has to be carefully worked out in advance to ensure the 
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adequacy of the samples for the information concerned. Statisti- 
cians or epidemiologists with experience of sampling methods 
should be consulted, because wrong samples will inevitably lead 
to false information, making valid judgement impossible. 


46. At this stage, it is necessary to plan the implementation 
of the survey in detail, including the types and number of staff 
required, the duration of the data-gathering and analysis periods, 
any travel required, data processing requirements, time schedules 
for completion of the survey, and the budgetary implications. The 
procedure for information analysis should also be planned in 
advance, and the full involvement sought of those engaged in the 
implementation of the programme or project being studied. For 
surveys of long duration, periodic reports may be required from 
those involved in order to monitor the implementation of the 
fact-finding procedures. 


47. Finally, if possible, a pilot run should be made before any 
large-scale survey is carried out in full. 


48. Before the information assembled is actually used, it should 
be reviewed in relation to its pertinence for the steps in the evalu- 
ation process concerned. Wherever justified, it should be pre- 
sented in tabular or graphic form in order to facilitate its systematic 
analysis. 


49. The consideration of relevance relates to the rationale for 
having the health policies, programmes, activities, services and 
institutions concerned. The following questions can be asked: 


(1) Are the health policies relevant to the country’s overall 
social and economic policies? 


(2) Are the programmes justified in social terms, i.e., are 
they aimed at giving effect to accepted health policies for solv- 
ing problems of high social relevance? 
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(3) Are the activities of the programme clearly related to its 
objectives? 


(4) Are the services and institutions concerned clearly re- 
lated to the attainment of defined national health goals or to 
the implementation of defined priority health programmes? 


50. The following list of criteria may be of use for further 
assessing relevance. These criteria should be used selectively. 


(1) There is a strong basis for the programme, service or 
institution... 

(a) because of its social relevance, e.g.: 

— itis directed towards defined national health goals 


— it contributes directly and significantly to the improve- 
ment of the health of the population concerned 


— it uses methods that can be applied and afforded now 
by the countries concerned 


— it responds to well-defined community needs 

(b) because the effect of not having the programme, service 
or institution would be: 

— to cause serious aggravation of the problem 


— that no other programme, service or institution would 
provide an alternative service to deal with the problem 


— other (specify) 
(2) There are strong reasons for ending the programme, 
service or institution because: 


— the problem has ceased to be of major public health 
importance or it is recognized that the programme is 
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yielding greatly diminished returns in relation to the 
efforts being expended 


— other (specify) 


If the results of the evaluation show that the policy, 
programme, service or institution is not relevant, recommend 


modification or, if necessary, termination 


51. The assessment of adequacy takes place with respect both 
to policy formulation and to programme formulation. In relation 
to policy formulation it includes assessing whether problems have 
been clearly defined; and, during programme formulation, whether 
the programmes have been adequately formulated. 


52. Assess whether problems have been clearly defined, ¢.g., 
the extent and severity of a disease, the populations affected or 
at high risk, the population to be covered by a service, the man- 
power problem for whose solution a training institution is being 
used. The list of criteria below may be of help for such an assess- 
ment. It should be used selectively and does not pretend to be 
exhaustive. 


53. The problem is of major public health importance 
(1) Severity 
— leads to high mortality rate 
— leads to high morbidity rate 
— leads to high disability rate 
— causes impairment of learning or work potential 


— causes impairment of growth and development 
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— endangers fetal life 
— other (specify) 
(2) Frequency of occurrence 
— rarely 
— occasionally 
— frequently 
— always 
(3) Related implications 
— has adverse political implications 
— has adverse social or economic implications 
— has adverse demographic implications 
— other (specify) 
(4) Distribution of problem 
— nationwide 
— regional or provincial 
— local 
(5) Inequity of health resources 


— there is an inequitable distribution of health resources, 
for example, a disproportionate share of health care 
resources (money, manpower, equipment, facilities, 
etc.) 


— other (specify) 


If the results of the evaluation of policy formulation show 
that the health problems have not been well defined, 
recommend redefinition of problems 
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54. Review if programmes have been properly formulated. 
Do they have long-term, medium-term or short-term objectives 
and targets? Have these been clearly defined? Are the organiza- 
tional and administrative responsibilities clearly defined? What 
approaches have been adopted for attaining the objectives, e.g., 
if the objective is to improve child health, has adequate attention 
been paid to appropriate nutrition, immunization, control of in- 
fections, and treatment of diseases and injuries? What methods 
and equipment have been selected and how were they selected? 
How have the programme components been assigned throughout 
the country? Has due consideration been given to manpower and 
financial requirements? 


55. The following criteria can be used as a check-list to help 
identify the above: 


(1) adequate provision has been made for the planning, man- 
agement and evaluation of the programme, and the informa- 
tion needed has been specified; 


(2) the objectives have been clearly stated, as far as possible 
in measurable terms; 


(3) a detailed plan of action with a time schedule (if appli- 
cable) has been established for the attainment of these objec- 
tives; 


(4) alternative approaches and methods were considered and 
it did not seem possible to arrive at a more economical ap- 
proach to attain the same objectives; 


(5) indicators and criteria have been selected for subsequent 
evaluation of the programme. 


If the results of the evaluation show that the programme 


has not been well formulated, recommend reformulation 
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56. The review of progress consists of an analysis of the 
efforts made and resources used as well as of the degree to which 
actual implementation complies with the planned implementation. 
The following unconnected examples are given by way of illustra- 
tion: 


— Compare the number of health personnel actually trained Review 
in a training institution in the course of a given period with progress 
the numbers planned. 


— Compare the actual population served by an institution with 
the population planned to be served (appropriate criteria 
have to be established to determine what is meant by a 
population being “served”, such as accessibility and degree 
of utilization of the service). 


— Compare the number of households supplied with piped 
drinking-water at a given date with the number planned to 
be supplied at that date. 


— Have equipment and supplies for the establishment of an 
institution arrived in time, and is there adequate provision 
for maintenance and resupply? 


— Have important events been achieved on schedule, e.g., the 
completion of the immunization of a given number of chil- 
dren against certain infectious diseases? 


— Compare funds spent with those budgeted. 


Only significant exceptions to the planned progress should be 
noted and, if possible, reasons should be given for significant 
deviation from the planned action. - 


57. The assessment of efficiency consists of an analysis of the Assess 
results obtained in relation to the efforts made and the resources efficiency 
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used. The question is: could these results have been obtained in 
better and more economical ways? A differentiation should be 
made between technical efficiency and cost efficiency. The follow- 
ing issues should be raised: 


(1) Operational level—assess whether the activities are being 
conducted at the right operational level, e.g., local, district or 
central. 


(2) Methods—analyse whether the methods employed are 
proving to be useful for the solution of the problem. 


(3) Manpower—analyse whether the best combination of 
available manpower is being deployed in sufficient numbers for 
the solution of the problem. Analyse the efficiency of the man- 
power in terms of skill and effort in comparison with what 
could be expected. 


(4) Finances—judge the adequacy of the financial resources 
made available for the implementation of the programme or 
the management of the service or institution. Make a cost analy- 
sis to assess how economically the results have been achieved. 


(5S) Facilities—review the suitability of the geographical 
location of buildings and the adequacy of buildings, vehicles, 
equipment and supplies, where applicable. 


(6) Collaboration—describe the results of collaboration with 
other social or economic sectors and institutions, voluntary 
agencies, bilateral and multilateral agencies, etc. 


(7) Managerial control—analyse factors not mentioned above 
such as the degree of adequacy of the time allotted for the 
implementation of the activities, the correctness of the sequence 
of activities, the appropriateness and timeliness of logistic 
support, and the degree to which major activity results were 
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being reached. Indicate problems encountered and describe 
corrective action envisaged or taken. 


(8) Cost-efficiency—this is an important aspect of the analy- 
sis of any programme, service or institution. 


58. Some examples follow by way of illustration. If the objec- 
tive of a programme is to train a given number of auxiliary nurse/ 
midwives each year, the learning methods, the types and numbers 
of faculty, the money invested in setting up training establish- 
ments and in running them, the suitability of the buildings and 
learning equipment and material, collaboration with the general 
education system, the time allotted to various components of the 
learning process and the sequence in which these components are 
being presented all have to be taken into consideration. Appro- 
priate educational criteria would have to be used, for example, 
to assess the usefulness of the learning methods. 


59. To give another example, if spraying indoors with residual 
insecticides has been selected as the control measure to reduce 
the incidence of malaria, the efficiency of the programme could 
be measured in terms of the degree to which houses are being 
correctly sprayed. The criteria used for this assessment are that 
the application of the insecticide has been made before the malaria 
transmission season; the insecticide has been applied at the rate 
of 2 g per m’; at least 95% of the inner surfaces of the houses 
treated have been properly sprayed; and 80% of all houses have 
been correctly sprayed. 


60. Another example relates to water supply. If the pro- 
gramme objective is to provide potable water for the total popu- 
lation, a social criterion for efficiency might be the ability of the 
population to establish and maintain this water supply by methods 
‘t can afford. Technical criteria would have to be used to assess 
whether the water supply is, in fact, safe. 
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61. The following example concerns an immunization pro- 
gramme. Suppose the objective of a programme is to reduce the 
incidence of certain infectious diseases of childhood. If these 
diseases can be controlled by the application of potent vaccines 
to a high proportion of the population at risk, the efficiency of 
the programme can be assessed by the potency of the vaccine 
employed at the time of its application and, in the final analysis, 
by the percentage of children at risk properly immunized. Tech- 
nical criteria are required to assess the immunization status of the 
children concerned. If it is not feasible to perform an assessment 
of immunization status, at least certain other criteria have to be 
applied, such as the number of children registered as having re- 
ceived the required doses of the vaccines concerned. 


62. One measure of the efficiency of a health service is the 
correct use of its component parts; for example, the use by the 
population of a health centre for preventive and simple curative 
services and use of the appropriate hospital only on referral from 
the health centre; another example could be the use of the most 
appropriate hospital in that it is nearest to the community con- 
cerned and provides clinical services at the right degree of speciali- 
zation. 


63. The following are a number of examples of the assessment 
of the efficiency of institutions: 


(1) Health centres—the comparison of the number of immu- 
nizations provided in a given period as compared with an 
accepted norm; the coordination between an immunization 
time schedule and the vaccine supply; the number of women 
provided with the required antenatal examinations over a given 
period of time as compared with an accepted norm. 


(2) Hospitals—the number of outpatients cared for per 
health worker. For inpatient care, the ratio of staff to beds; the 
percentage of bed occupancy; the average duration of stay by 
clinical department; the bed turnover rate; the number of in- 
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patient days for a given size of population; the number and 
types of laboratory or radiodiagnostic examinations for a given 
size of population; medical audit, e.g., the comparison with 
certain norms of the statistics for the care of a given condition, 
such as average length and frequency distribution of stay, types 
of care provided, results of care, the number of patients return- 
ing for the same condition, and the frequency of their return. 


64. The cost-efficiency of institutions should be measured in 
terms of the amount of service provided in relation to the costs. 
For example, the efficiency of including a trained nurse/midwife 
in the staff of a health centre without obstetric beds has to be 
measured in terms of the cost divided by the number of women 
provided by her with the accepted range of antenatal and post- 
natal care; the efficiency of having such a midwife for deliveries 
has to be measured in terms of the cost divided by the number 
of deliveries. The comparison of the cost-efficiency of different 
hospitals has to be assessed in terms of the costs divided by the 
total number of inpatients, but great care has to be taken in ar- 
riving at conclusions in order to take into account variations be- 
tween hospitals in the provision of different kinds of diagnostic and 
therapeutic inpatient and outpatient care. It is often more fruitful 
to assess the cost-efficiency of the same institution over a number 
of given periods of time. 


65. Quality of care is an important component of efficiency, 
but is usually very difficult to assess objectively. In all the above, 
it has been taken into account only in terms of medical audit in 
hospitals. 


66. The assessment of effectiveness should be mainly an Assess 
analysis of the attainment of objectives, expressed, if possible, in effectiveness 
terms of health problem reduction or an improvement of an un- 
satisfactory health situation. The definition of targets and output 
indicators during programme planning will greatly facilitate sub- 
sequent evaluation of effectiveness. 
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67. The following are a number of examples. If the objective 
of a programme in a country is to improve child nutrition, the 
output indicator could be the number of children whose nutri- 
tional status has improved following the introduction of the pro- 
gramme. If one of the targets set was to reach a satisfactory nutri- 
tional status for all preschool children by a certain date, the effec- 
tiveness of the programme could be assessed in terms of the per- 
centage of preschool children with an adequate nutritional status 
at that date. The indicators for assessment would be those that 
could be applied in practice in the country concerned for assessing 
nutritional status. 


68. To give a further example, if the objective of a programme 
is to reduce the incidence of malaria, the effectiveness of the 
programme will be measured in terms of the annual incidence at 
the time of evaluation as compared with the annual incidence at 
the beginning of the programme. The technical criteria required 
are those used for measuring the incidence of malaria. 


69. To revert to the example used in relation to the efficiency 
of a programme of immunization against certain infectious dis- 
eases of childhood (see paragraph 61 above), the ultimate mea- 
sure of effectiveness will be the reduction in mortality and mor- 
bidity rates for these diseases. 


70. In conformity with the above, assess the extent to which 
objectives appear to be attained and targets met. Identify, wher- 
ever applicable, reasons for failure to attain objectives or meet 
targets, and review the adequacy of any corrective action taken. 
Special attention should be given to problems encountered for 
which solutions are still sought. Analyse to the greatest extent 
possible the effect of the programme or project in attaining the 
objectives of the broader programme of which it forms a part. 
For example, if the objective of a programme for training mid- 
wives has been attained, has the natal and infant care been im- 
proved as measured by the increase in the number of child-bearing 
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women being attended by midwives and ultimately by a reduction 
in maternal and perinatal mortality? If the target for village water 
supply has been reached, has this helped to lead to a reduction 
in the incidence of diarrhoeal diseases? In addition, assess as far 
as possible whether the results of the programme have met the 
expectations of the community concerned and its leaders, whether 
local or national. 


71. The assessment of impact is the most difficult step in the 
evaluation process. The question that has to be asked is: even if 
the objectives of the programme are being attained, has the re- 
sult been an improvement in the overall health and socioeconomic 
situation and in the quality of life? For example, the provision of 
water supply could have a beneficial effect on the health of the 
country in many ways, yet it could have a detrimental effect if 
carried out alone without an accompanying drainage system. The 
control of specific infectious diseases in infancy could be con- 
sidered a success when infant mortality is reduced, but the cor- 
responding increase in the number of children in the community, 
if unaccompanied by appropriate agricultural development and 
nutritional programmes, could lead to problems of malnutrition. 


72. In order to draw conclusions, summarize the objectives, 
approaches, methods and results of the programme. For 
services and institutions, show the relationship between, on the 
one hand, the functions they are performing and the approaches 
and methods being used to fulfil these functions, and on the 
other hand, the programmes they are intended to deliver, and the 
results of the performance of the functions. 


73. Summarize the supporting information which leads you 
to conclusions concerning policy and programme relevance, prob- 
lem definition, the adequacy of programme formulation, the 
progress being made, the efficiency of implementation, the effec- 
tiveness of the programme, and its overall impact. Indicate per- 
ceived achievements as well as major problems encountered. 


Assess impact 


Draw conclu- 
sions and 
formulate 
proposals for 
future action 
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74. When preparing this summary, make sure that the com- 
ments of all those involved in the evaluation process are taken 
into account and are fully discussed with the individual responsible 
for the programme, service or institution being evaluated. 


75. Draw conclusions in the light of the information sum- 
marized as above. These conclusions should provide a basis for 
modifying programmes, or reprogramming if necessary. Formulate 
proposals for future action as appropriate, e.g., to redefine prob- 
lems which were improperly presented; to redesign programmes 
inadequately formulated or to accelerate their progress; to re- 
define the functions and structures of an institution or service; 
to increase the budget, etc. (for examples of the types of decisions 
that have to be taken, see paragraph 39(5) above). Suggest whether 
the programme or project should continue, be modified or be ter- 
minated. Specify any necessary modification to the objectives, tar- 
gets, methods and techniques applied, as well as to human and 
financial resources to be deployed and the time required for the 
intended change. Propose options for choice of modification. 
These proposals should be included in reports to be addressed to 
the bodies or individuals mentioned in paragraph 39(6) above. 


IV. The evaluation of the 
managerial process for national 
health development 


76. From time to time, national health authorities may wish 
to evaluate the managerial process for national health develop- 
ment that they are using. Their managerial process would thus 
become the subject for evaluation. In the absence at this stage 
of suitable indicators and criteria for such evaluation, it is sug- 
gested that use be made of pertinent questions. The following 
illustrates how the evaluation process might be applied to evaluate 
the country’s managerial process, relying on the use of such 
questions. 


77. The particular subject for evaluation could be the extent 
to which the managerial process is achieving its purpose, if at all. 
Another particular subject could be the cost of the process in 
relation to its effectiveness or the adequacy of a particular compo- 
nent of the process, for example, programme budgeting. 


78. A question to be asked is if the managerial process is 
relevant to the political, social and economic situation, and to the 
overall socioeconomic planning process in the country. This might 
include comparing the planning methods used in the managerial 
process on the one hand with those used for planning SOCIO- 
economic development in general on the other hand, to see if 
the two are compatible. It might also include comparing the data 
envisaged for submitting health programme proposals to the 
government with the deadline specified for the country’s five- 
year plan or annual budget. . 


79. The assessment of the adequacy of the managerial process 
could include investigating whether the process has been spelled 


— 


Specify the 
subject for 
evaluation 


Verify relevance 


Assess adequacy 


45 


Health programme evaluation 


Review progress 


Assess efficiency 


Assess 
effectiveness 


46 


out sufficiently to ensure the application of the whole range of 
managerial activities required to implement the national strategy 
for health for all. Has the process been adapted to the capacity 
of those who have to apply it? Is it clear to them how they should 
apply it? Have guidelines and accompanying training material 
been prepared, and are people being trained in their use? Are all 
important disciplines, areas and authorities adequately participat- 
ing? Have appropriate permanent mechanisms been established, 
including community involvement at local, intermediate and 


central levels? 


80. The review of the progress of the managerial process 
could include considering whether it has developed and is being 
applied at the pace required, whether it is being applied as con- 
ceived, and whether it is being used throughout the whole country, 
throughout certain parts of the country only, or for specific 
programmes, services and institutions. 


81. To assess the efficiency of the managerial process it is 
necessary to consider whether it could be carried out more eco- 
nomically and in a shorter time. This includes assessing whether 
the process could be carried out with fewer people, with less 
skilled people, and with greater community involvement. 


82. The effectiveness of the managerial process has to be 
assessed in the final analysis from its results. For example, has 
the process given rise to clear health policies in support of health 
for all by the year 2000, to a well-defined national strategy to 
give effect to these policies, and to appropriate plans of action 
with adequate budgets to implement the strategy? Have priority 
programmes been adequately formulated, and have the services 
and institutions for delivering these programmes been properly 
designed? Have the programmes been properly integrated into 
the general health system, starting with primary health care 
and continuing through the subsequent levels of the health system 
as required? Are programmes being efficiently implemented, 
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and are the services and institutions for delivering them being 
efficiently run? 


83. Has evaluation been properly integrated into the various 
components of the managerial process, in practice as well as in 
theory? Has this led to modifications in programmes, services and 
institutions, and to reprogramming if necessary? Is the informa- 
tion support adequate in that information requirements have been 
carefully identified, the information required has been forth- 
coming, and the data collection and analysis have been carried out 
in the most economical way? 


84. In the light of the above, draw conclusions and formulate 
proposals for future action, by indicating how far the managerial 
process is relevant to the country’s managerial process for socio- 
economic development in general, and suggest ways of increasing 
this relevance if necessary. Judge whether the process is clear, 
well understood by those who have to apply it, and properly ap- 
plied; and if not, whether this is due to lack of the right kind of 
people, guidelines, training, funds, or appropriate mechanisms. 
Indicate corrections required, for example, with regard to the 
people applying the process; mechanisms for ensuring its proper 
development and application; training; the manner of application; 
and the pace and location of application. Suggest ways of improv- 
ing efficiency and, if the process is not sufficiently effective in 
producing results, indicate the aspects that are in need of improve- 
ment as well as ways of bringing about such improvements. 
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